Guidlines for running the CCTV ML notebook 7

There are two models that we use for prediction: structural_rectangular_fix and infiltration_rectangular_fix.

Config section

In the config section, you can set the utilities to run the lightgbmodel by adding the utility name to the list_utility list.

You can choose between predicting structural and infiltration defects, by setting 'y=I' for infiltration defects and 'y=Structural_defect_I'.

You can also set the directory paths in this section.

The .py file has a menu that it creates in the CLI to set the config variables.

General guidlines:

Shapefiles get saved in unionsine1/VS_Research/CCTV/DNV/GIS/shapefiles/final_model/infiltration_prediction (structural_prediction)

Util functions section has to be run.

prepare_data() has to be run. custom_train_test_split() has to be run

Oversample and unsampling training data:
resampling_training_data has to be run Training and Prediction Function:
Run the section custom_f2_score can be skipped data_roc_curve can be skipped save_metric_scores_of_the_model can be skipped gridsearch_best_params can be skipped catboost_param_grid can be skipped

Skip the Main section but run the cells above it.

skip LOAD_TO_BE_PREDICTED_FROM_CSV for Polar Coordinates (since we are sticking with rectangular coordinates).

FORMAT to_be_pred_df SO IT CAN BE PASSED INTO MODEL - Run the code cell tht has this comment.

Run the cell whcih has the series = ['AC', 'CP', 'PE', 'PVC', 'RCP', 'SP', 'VCP', 'XXX', 'ZZZ'] field to create categories.

Tasks to do:

Save prediction files up to 3 decimal places (pred_I column) df_pred_final, df_pred_train, df_pred_test

df_pred_final.to_csv(...) add a timestamp to the end of the file

Fixing pipe lengths email by Jake:

All,

I've fixed the negative lengths and I'm attaching the two new input files needed. I'm also uploading them to sharepoint.

Quick overview of the process: Open SanMains.shp in qgis Open attribute calculator Select "update existing attribute" and select the "ASBLENGTH" Type in the formula "$length" and click okay. Save the changes to the file. Open the DBF with Excel and save as CSV. Import this CSV to SQLite. Update the two tables in SQLite corresponding to the two relevant input files (input_DNV_A_B_C_version_3(row_per_pipe_id) and DNV_SanMains_not_train_test) with the newly updated lengths. Export those two tables as CSV.

Here are the update queries I used, for reference:

UPDATE "inputDNV_A_B_C_version_3(row_per_pipe_id)" AS b SET pipe_length = g.ASB_LENGTH FROM SanMain_NEG_LEN_FIXED AS g WHERE b.pipe_id = g.ASSET_ID;

UPDATE DNV_SanMains_not_train_test AS b SET ASB_LENGTH = g.ASB_LENGTH FROM SanMain_NEG_LEN_FIXED AS g WHERE b.ASSET_ID = g.ASSET_ID;

Article to use CUDA for lightGBM: https://stackoverflow.com/questions/60360750/lightgbm-classifier-with-gpu